Video Content Representation for Shot Retrieval and Scene Extraction

+ The work reported in this paper was mainly performed when this author was working at Microsoft Research, China as a research intern. In this paper, we present a novel scheme on video content representation by exploring the spatiotemporal information. A pseudo-object-based shot representation containing more semantics is proposed to measure shot similarity and an force competition approach is proposed to group shots into scene based on content coherences between shots. Two content descriptors, color objects: Dominant Color Histograms (DCH) and Spatial Structure Histograms (SSH), are introduced. To represent temporal content variations, a shot can be segmented into several subshots that are of coherent content, and shot similarity measure is formulated as subshot similarity measure that serves to shot retrieval. With this shot representation, scene structure can be extracted by analyzing the splitting and merging force competitions at each shot boundary. Experimental results on real-world sports video prove that our proposed approach for video shot retrievals achieve the best performance on the average recall (AR) and average normalized modified retrieval rank (ANMRR), and Experiment on MPEG-7 test videos achieves promising results by the proposed scene extraction algorithm.